{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "The full extended_concepts.ipynb will be uploaded by Wednesday the 18th, with coded examples of the below methods to utilise FlowTransformer to measure the following concepts. For now we include a description of how to use FlowTransformer to achieve these measurements."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Temporal Bias:\n",
    "- Run FlowTransformer in sequential mode over 50% of the dataset using the train method of the Keras model in a custom  training loop\n",
    "- Call FlowTransformers evaluation method on the remaining 50% of the dataset\n",
    "- Repeat for varied spacing between train and test dataset (going to 40% of the start of the dataset and 40% of the end, then so on) to sample different spacings of data\n",
    "- Plot a graph of distance from the training to the test data, versus the evaluation performance, to demonstrate temporal bias if any\n",
    "\n",
    "This will demonstrate if the time window of data being learned, being separate from the window of data when the model is being actually applied, has any influence on the performance of the model."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Transferability:\n",
    "- Run the train method using one dataset, or one class from a dataset\n",
    "- Measure the evaluation accuracy on other datasets or classes\n",
    "- Plot a graph showing for each alternative class or dataset, how the F1 score changed from training data versus the alternative data to demonstrate generalisability\n",
    "\n",
    "This will demonstrate if the classes of data being learned, in the datasets being utilised, provide adequate training for the model to transfer to new datasets or classes."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [
    "## Concept Drift:\n",
    "Similar to Transferability, we can perform the same analysis, but gradually phasing between two datasets we know come from disparate networks, to simulate a change in behavior over time. We provide an implementation of this dataset intermeshing as part of a custom evaluation loop."
   ],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   }
  },
  {
   "cell_type": "markdown",
   "source": [],
   "metadata": {
    "collapsed": false
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}
